Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix resolved (template) values in case of failure #4532

Merged
merged 2 commits into from
May 30, 2024

Conversation

Flamefire
Copy link
Contributor

@Flamefire Flamefire commented May 14, 2024

When the value cannot be resolved usually the original value is returned.
However if it contains % signs there will be an escape step and that escaped value is returned.

That makes it impossible to have values which can only later be resolved, like
cd %(startdir)s in extensions which would become
cd %%(startdir)s on the first resolve-attempt.

This is impossible to "undo" in the current framework hence we must take care of this case.

I also did some related changes:

  • pylint detected some cases where dict.items should be used
  • The escaping/unescaping of template values as documented in resolve_template was not tested

Recent actual failure was using cp %s %(builddir)s/archives as the extract command in a component. I'd argue that should work as expected although there is a workaround: Escaping the %(builddir)s template. But that kind of requires to know how many times EasyBuild will try to resolve templates.

Flamefire added 2 commits May 14, 2024 16:13
When the value cannot be resolved usually the original value is
returned. However if it contains `%` signs there will be an escape step
and that escaped value is returned.
That makes it impossible to have values which can only later be
resolved, like `cd %(startdir)s` in extensions which would become
`cd %%(startdir)s` on the first resolve-attempt.
@boegel
Copy link
Member

boegel commented May 23, 2024

@Flamefire The case from easybuilders/easybuild-easyconfigs#20119 that happens to work now should also be taken into account.
Although that happens to work by accident almost, can we somehow prevent from breaking that use case where %(builddir)s is being escaped to avoid trouble with the %s that accompanies it?

@Flamefire
Copy link
Contributor Author

can we somehow prevent from breaking that use case where %(builddir)s is being escaped to avoid trouble with the %s that accompanies it?

I'm not sure we can and don't think we should. #4516 might also have avoided the issue we had as it avoids resolving where we can and allows failures at places where they might happen. Haven't checked but I guess there it won't try to resolve the extension values when reading them. And with this change it won't matter anymore how often EB tries to resolve templates until it succeeds: The value stays unchanged.

Additionally we have:

# '%%(name)s' -> '%%(name)s', and after templating with {'name':'x'} -> '%(name)s'

According to this no-one should expect that workaround to work, should they? And the idea was explicitly that things like %(builddir)s %s work, i.e. the %s is not tried to be resolved by EB during template steps, see the test for the single %s (which works now too): https://github.com/easybuilders/easybuild-framework/pull/4532/files#diff-bf1c3722aa26c6569dd59d4e4a92f16c88bf688276708beb9f09df6e40288713R3662

Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegel boegel merged commit 431adef into easybuilders:develop May 30, 2024
37 checks passed
@Flamefire Flamefire deleted the fix_partial_template_resolve branch May 30, 2024 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants